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The disclosure provides improvements in effecting the 
routing function in a communication network operating 
under a distributed control protocol. Selected nodes 
(control nodes) have extended memory and computing 
capabilities and each maintains a topology data base, 
other nodes (ordinary nodes) have more limited mem- 
ory and computing capabilities and maintain informa- 
tion respecting solely local topology. The method of 
the invention provides for maintaining the topology 
data bases in the control nodes current in the face of 
changes in the network, provides for the selection of a 
particular control node from which a particular ordi- 
nary node will obtain necessary routing information and 
establishes and maintains communication between the 
ordinary node and the selected control node. Control 
nodes identify adjacent control nodes, and upon deter- 
mining changes in network status, such network status 
changes are communicated to the adjacent control 
nodes. When a routing decision is required at an ordi- 
nary node, the ordinary node refers to the single control 
node co-existing in the ordinary node's domain, from 
which to obtain information necessary for routing pur- 
poses. 

16 Claims, 19 Drawing Figures 
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3. Establishing and maintaining communication bc- 
AUTOMATIC UPDATE OF TOPOLOGY IN A tween the ordinary node and the selected control 

HYBRID NETWORK node. 

It should be understood that the solution to these prob- 
DESCRIPTION 5 lerns must survive a dynamic environment in which 

links and/or nodes (both ordinary and control) may fail 

1. Technical Field hence become unavailable) or recover from such a 
The present invention relates to communication net- hence become available). 

works and specifically improvements in effecting the 

routing function of a network using distributed control. 1Q The Ownership Concept 

2. Background Art When an ordinary node wants to communicate to 
In large mesh connected communication networks, another node m ^ network, as indicated above, it may 

proper routing of messages so that they effiaentty reach haye tQ gQ tQ ft node to obtain a route . i n addi. 

a selected destination node presents many difficulties. ^ jf m ordinary node detects a change in its adjacent 
The function is discussed by Heart et al m "The Inter- {$ top61ogV( that information must be communicated to 
face Message Processor for the ARPA Computer Net- node so that the information may be re- 

work" appearing in Vol. 40, AFJPS Conference Proceed- fleeted in the topology data bases. Thus any time an 
ings (1972) at pp. 551-567, Frank et al in "Topological ordinary node is up, a "session" called an ownership 
Optimization of Computer Networks", in Proceedings of ^ion should be established between the ordinary node 
the IEEE, Vol. 60, pp. 1385-1396 (Nov. 1972), Rudrn in 20 and node . However, as described below 

"On Routing and 'Delta Routing"* in IEEE Transac- ^ ^fe^ c f ^ ownership session is not essential at 
turns on Communications, Vol. COM-24, pp. 43-59 (Jan. times. The procedures in the various nodes are biased 

1976), and Dayies et al in Computer Networks and Their t0 e^jp^e an ownership session if at all possible. 
Protocols (John Wiley & Sons, 1979), see Chapter 3 and yh e collection of ordinary nodes connected at a 
in particular pages 109-1 14. Two competing principles 25 g ysR by sessions to a particular control node, N, is 
are centralized routing and distributed routing, in the referred to as N*s domain and each such ordinary node 
former all routing is effected by a single central author- ^ said to ^ owne d by N. Whenever a new ordinary 
ity, whereas in the latter approach route control is dis- node comes up, an ownership session to some control 
tributed throughout the network. The present invention nodc sn0 uld be established by the following procedure, 
is directed at improvements in distributed control of the 30 ^ the ordinary node will thereby join the correspond- 
routing function. jng domain. 

A communication network typically consists of a The procedure for setting up ownership is in princi- 
plurality of nodes, and communication links (hereinafter p | e ^ follows: when ordinary node i first comes up or 
"links**) interconnecting the nodes. Nodes connected by j oses its owner because of an outage of a pre-existing 
a single link are considered adjacent The nodes can act 35 ownership session, the ordinary node i informs its 
as an information accepting location (origin node), in- neighbors or adjacent nodes (nodes directly connected 
formation sink location (destination node) or an inter- to no( j c \ 9 whether control nodes or ordinary nodes) 
mediate node in passing a message from the origin to the aDOUt this fact. If the neighboring ordinary nodes are 
destination. Thus the routing function, to be effected, owned, they communicate the information to their own 
requires an understanding of the topology of the net- 40 owners, who in turn attempt to establish an ownership 
work. Especially in large networks, the topology of the session with the ordinary node L That attempt is imple- 
network is far from constant, the routing function must merited by the owner's transmitting a request for own- 
be capable of operating in an environment wherein ership (or an invitation) to the ordinary node i. The 
nodes are being added and deleted. Such deletions or rQ ute for such message is first the ownership session 
additions may be the result of expansion or contraction 45 between an NC and the neighboring node and second 
in the network and/or communication failures in a node the link between the neighboring node and ordinary 
or a link. node i. The ownership request message which first 

Because the information describing the topology of arrives at the ordinary node i is selected as the success-, 
the network can be extensive, we choose to employ two ml one. If on the other hand a neighbor j of the ordinary 
different types of nodes in the network, a control node 50 node i is not owned, j saves knowledge of the fact that 
(NC) which has extended memory and computing capa- the ordinary node i is not owned, so that subsequently if 
bilities, and an ordinary node (NNC) which has more j becomes owned, j transmits to its new owner the fact 
limited memory and computing capabilities. To follow that the ordinary node i is unowned. Note that a domain 
the distribution of the resources in the network, we may well include more than the nodes adjacent the 
propose that only the control nodes maintain a topology 55 control node. 

data base (indicative of the present status of the net- Tfae Virtual ^etwc^ 

work) and that when an ordinary node requires routing 

information, that information be acquired from a con- The various topology data bases of the network need 
trol node. accurate information about the current status of the 

The present invention is particularly directed to the 60 network. Consequently, when a topological change 
solution of three problems that arise in this environ- occurs in the form of a failure or recovery of nodes and 
ment . links, such information must be transmitted to all con- 

1. Maintaining the topology data bases current in the trol nodes. In order to achieve this with a relatively 
face of changes in the topology or performance small load on the network, we define a virtual network 
characteristics of the network resources, 65 of control nodes. 

2. Selection of a particular control node from which The nodes in this virtual network will consist of all 
a particular ordinary node will obtain necessary the control nodes. The links of the network (virtual 
routing information, and links) are sessions in the original network connecting 
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certain pairs of control nodes. Pairs of control nodes changes in topology and/or weight must be reported to 

connected by virtual links are called virtual neighbors the control node so that the topology data base of the 

and the broadcast of topological information will prop- control node will provide an accurate reflection of the 

agate only on the virtual network (once the information present status of the network. This is performed as fol- 

reaches the first control node). S lows: whenever a node senses a change in an adjacent 

The topology of the virtual network, namely what link or when an Ordinary node gets a new owner, the 
control node should be defined as virtual neighbors and node reports the status of all its adjacent links and their 
connected by virtual links, is essentially arbitrary, so weights to the owner, the message including a time 
long as the virtual network remains connected. Our stamp from the transmitting node. Whenever a control 
choice for the topology of the virtual network is as 10 node receives such a message from an owned node k, 
follows. The domain of control node N is defined as the the control node updates the topology by replacing the 
collection of all nodes owned by N. Two control nodes list of links adjacent to the node k in its topology data 
will be connected by virtual links if their corresponding base with the new list The topology update is per- 
domains are contiguous (there is at least one link con- formed in the same manner when the control node itself 
necting a node in one of the domains to a node in the 15 senses an adjoining topological or weight change. After 
other domain). This choice guarantees that all control the control node has updated its own data base, it pro- 
nodes within a connected network will be connected in ceeds to inform all other control nodes about the new 
a virtual network. Thus, the virtual neighbors are virtu- status via a broadcast protocol by sending a broadcast 
ally adjacent (connected by a single link). message with the information to each neighbor on the 

We now describe how the control nodes know when 20 virtual network. Whenever a node on the virtual net- 
and where to set up sessions to their virtual neighbors. work receives such a broadcast, it checks its topology 
Each control node knows the identity of other control entry for the node k. If the time stamp in the message is 
nodes that should be its virtual neighbors through the less than or equal to the current time stamp stored for 
ownership protocol. In particular, each control node the node k, the broadcast message is discarded. Other- 
learns the name of the owner of any node contiguous to 25 wise the receiving control node changes its topology 
its domain. To physically set up the session, additional table entry and proceeds to transmit the identical mes- 
protocols are needed. If the control node would have sage to all of its virtual neighbors except the virtual 
enough topology information to determine an entire neighbor from which it received the message. In certain 
route to the target node, it could simply set up the route. cases, parts of the virtual network may not get certain 
However, in certain cases (for example at network start 30 information from the broadcast protocol. This may 
•up), the control node does not in fact have sufficient happen if the network becomes temporarily discon- 
route information. This occurs for example when a first nected or due to delays in establishing virtual links. This 
- control node finds out that a second control node must is overcome by requiring every control node to ex- 
now become a virtual neighbor as a result of two ordi- change topology tables with each new virtual neighbor 
nary nodes in the different respective domains becom- 35 or with a virtual neighbor to which it temporarily did 
ing connected by a new link, or though a change in not have a virtual link, and to broadcast the parts of the 
ownership. In that case, the first control node may have topological data base that are not identical. 

been previously disconnected from the second control ™ ' . , 

node and does not have sufficient information to deter- BRIEF DESCRIPTION OF THE DRAWINGS 
mine the route to the second control node. Neverthe- 40 The present invention will now be described in fur- 
less, the definition of the virtual network insists that ther detail in the following portions of the specification 
somehow the first and second control nodes must get when taken in conjunction with the attached drawings 
into a session to share information on their previously in: 

disconnected network components. To overcome this FIG. 1 is a block diagram of a typical hybrid commu- 
problem, a route can be established between the first 45 nication network in accordance with the invention; 
and second control nodes by concatenating the different FIGS. 2 and 3 are block diagrams of a typical control 
ownership sessions. For example, if you assume that a and ordinary node, respectively; 
first ordinary node is in the domain of the first control FIG. 4 illustrates schematically interconnection of 
node, and a second ordinary node is in the domain of the domain conforming to the network of FIG. 1; 
second control node, then necessarily the first control 50 FIGS. 5-11 are flow diagrams of the processes per- 
node has an active (ownership) session with the first formed at an ordinary node in response to various con- 
ordinary node, and the second control node has an ditions, in accordance with an implementation of the 
active (ownership) session with the second ordinary invention; 

node. Furthermore, the requirement for communication FIGS. 12-18 are processes effected at a control node 

between the first and second control nodes arises as a 55 in response to various conditions in an implementation 

result of a new link between the first and second ordi- of the present invention; and 

nary nodes. Therefore a route between the first and FIG. 19 shows the routing process at a typical ordi- 

second control nodes exists by concatenating the route nary node. 

of the ownership session between the first control node . __ ^^™™™~^ 

and the first ordinary node, the new link between the 60 DETAILED D ^^NOT PREFERRED 

first and second ordinary nodes, and the route of the EMBODIMENTS 

ownership session between the second control node and A communication network includes nodes and links, 

the second ordinary node. and a communication network with which this inven- 
tion is concerned is an arbitrary mesh connected hybrid 

65 network. The network is a hybrid in that it includes at 

The status of the network includes both its connectiv- least two different types of nodes, control nodes (NC) 

ity and an efficiency or capacity factor (weight) as- each of which has extended memory and computing 

signed to each of the links. Information reflecting capabilities, and ordinary nodes (NNC), with more 
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limited memory and computing capabilities. Both the gathers local topology information for its link status 

control and ordinary nodes are general or special pur- table 208. 

pose computers with memory for data bases, logic and The communication component 101 or 201 has the 

communication capabilities. In addition to executing function of accepting a message for transmission from a 

network operations, these computers may also have 5 particular application and transmitting the message on a 

user applications running, which user applications may link in an appropriate format for transmission. It also 

or may not have any application to the network. All the has the function of receiving a message in the transmis- 

nodes in the network play some part in routing some sion format and altering the format to make the message 

messages from an origin to a destination node. The available to a particular application. The communica- 

minimal function required of ordinary nodes is, on re- 10 tion component 101 or 201 also selects a particular link 

ceipt of a message destined for a node other than itself, based on information from the routing function 102 or 

to acquire sufficient information to select a particular 202. The communication component is also responsible 

link associated with a particular neighboring node on for reporting on changes in status of any connected link, 

which to transmit the message. The control nodes are The communication component 101 or 201 is wholly 

the resource to which an ordinary node applies in order 15 conventional and is not described further herein, 

to obtain information necessary for routing. The man- The routing function 102 or 202 accesses the associ- 

ner in which an ordinary node selects a particular con- ated routing table 107 or 207 to determine a particular 

trol node from which to obtain this information, the link on which to transmit for the message to reach a 

manner in which the changing status of the network is desired destination. Inasmuch as the routing function is 

reflected in a data base maintained in a control node, 20 wholly conventional, aside from the manner in which 

and the manner in which information contained in the routing data is obtained, it is not further described. The 

data bases of plural control nodes is distributed, is par- routing table 107 or 207 is merely a storage area into 

ticularly described hereinafter. which information is written and from which selected 

FIG. 1 depicts a typical communication network information may be accessed by the routing function, 
including control nodes, ordinary nodes and links inter- An example of a routing table is shown in Table 1. 
connecting the same. The links have certain characteris- The topology data base 105 and link status table 208 
tics associated with them and the degree to which any are additional storage areas into which selected infor- 
link has the characteristics is collectively referred to as mation is written and from which selected information 
its weight (W). Changes in the characteristics of a link 3Q may be accessed. The collection of topology data bases 
are reflected as a corresponding change in its weight. of all NCs is the source of information from which 
FIG. 1 shows control nodes A, B, C, and ordinary routing tables 107 and 207 are written. The link status 
nodes a, b, c d (control nodes are designated in upper table 208 is the source from which information is ex- 
case as well as by the designator NC, ordinary nodes are tracted to form the basis of messages to owner NCs. 
designated in lower case and the designator NNC). The 35 The Local Topology Monitor 103 or 203 is the corn- 
purpose of a network such as is shown in FIG. 1 is to ponent of a node which maintains the status of the corn- 
allow users to exchange information. That exchange munication links from this node to all other directly 
can be in the form of packets. Thus in FIG. 1 an applica- adjacent nodes. Initially, it obtains knowledge of the 
tion in NC A can exchange packets with an application existence of communication links through operator 
in NNC b. 40 definition or by having intelligent communications 

Since NC A is not directly connected to NNC b by adapters making themselves known during a power-up 
any single communications link, any packets originated sequence. Subsequent additions or deletions of commu- 
within NC A destined for NNC b (and vice versa) must nication links are made known to the Local Topology 
traverse other nodes in the network. Each intelligent Monitor in similar fashion- 
node in the network (both NCs and NNCs) are capable 45 Once active with definitions of the local communica- 
of forwarding messages not originated by them to the tion links available, the Local Topology Monitor is 
correct destination. This is accomplished through a responsible for beginning, monitoring, and tenninating 
routing table in each node. communication with adjacent nodes. The beginning 

Each NC in a network maintains a topology data phase of communications is highly dependent on the 

base, which contains its understanding of the nodes and 50 ^ technology. Upon operator request or as a part of 

connections in the network. From this topology data normal operation for that link, the Link Topology Mon- 

base, an NC can determine the series of nodes (called a itor actively seeks to establish a connection to an adja- 

path) between any two nodes which could support the cent node. Successful connection to an adjacent node 

exchanges of packets. Once a path has been determined, causes procedures detailed below to occur, 

the routing tables in each node along the path are up- 55 In the monitoring phase after successful connection 

dated to support routing packets along the path. FIG. 2 has been established to either an ordinary node or a 

depicts the typical structure of a NC. Each NC has control node, the Link Topology Monitor constantly 

communications component 101, packet routing func- examines the characteristics of each link to detect 

tion 102 which uses a routing table 107, and optional changes in link weight An example of a cause for such 

applications 106. In addition, the topology data base 105 60 a change might be a significant increase in transmission 

is maintained from the status of the local topology mon- delay over a link due to heavy traffic. Weight changes 

itor 103 and from information from other nodes in the for a link cause procedures detailed below to occur, 

network (as will be described). In the terminating phase, link communication has 

FIG. 3 shows a typical NNC Each NNC retains only ceased. The cause for cessation could range from an 
local information about its own links and adjacent nodes 65 orderly shutdown by the adjacent nodes to a sudden 

in a link status table 208. Like a NC, a NNC has a com- failure of the link. The loss of a communication link and 

munications component 201, packet routing function possible ownership session losses cause procedures de- 

202, and a routing table 207. However, the NNC only tailed below to occur. 
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All changes— activation, weight changes, termina- Multiple NCs are supported, relieving the network of 

tion — are retained by the node. An ordinary node re- having any single node whose failure would prevent 

tains this information in its Link Status Table, whereas new communication paths from being established, 

a control node retains this information in its broader SUMMARY OF HYBRID NETWORK 

Topology Data Base. 5 OPERATION 

The domain topology monitor 104 is similar to the 

local topology monitor 103 or 203 except that its scope Maintaining Local Topology 

is the entire domain rather than merely directly adja- Each no d e monitors its own links. When a new link is 

cent topology. activated, a node exchanges and retains ownership in- 

The procedures described below (FIGS. 5-18) are 10 formation about its new neighbor, 

implemented m the local topology monitor 103 and the ■ 

domain topology monitor 104. Establishing Ownership 

Since an NNC does not maintain a topology data An NC is owned by itself, 
base, it is unable to compute all complete paths which An NNC establishes an owning NC as follows: When 
may be required for routing. Therefore, each NNC NNC i first comes up or loses its owner because of 
must associate itself with a NC in order to request path outage of the ownership path, it informs its neighboring 
information for establishing communication between its nodes of this situation. If a neighboring node j has an 
applications and the applications in another node. The owning NC, it communicates this information to its 
NC to which an NNC associates itself is said to be the 2Q owning NC, who in turn attempts to establish an owner- 
owner of that NNQ and the communication of topol- ship path at NNC i. The NC whose attempt arrives first 
ogy information between an NNC and its owning NC to NNC i is selected as the owner of NNC i. If on the 
uses an ownership path previously established by that other hand, the neighboring NNC j is not owned, NNC 
NC. An NC uses its own topology data base for its path j saves the knowledge that NNC i is not owned. Subse- 
determination. 25 < l uent *y» when it becomes owned, NNC j notifies its 

The NC and the associated NNCs (if any) which use new owning NC that NNC i is not owned. The NC that 

its topology data base are called the domain of that NC. owns NNC j can now attempt to establish ownership of 

The arbitrary topology of FIG. 1 shows domain bound- NNC i. If and when NNC i becomes owned, it informs 

aries (dotted) for one possible association of NNCs with its neighbors (j) of its owner. If j is owned it reports on 

NCs. 3Q i's owner to j's owner. 

The NCs and their associated domains form a virtual Acquiring Domain Topology 

network of NCs, where the nodes of the network are . , . 

the NC domains. A connection between nodes of this Ml chan S es . m topology and ail changes in connec- 

virtual network exists when there is an established com- tion characteristics withina domain must be recorded 

munication path between NCs for the purpose of ex- 35 by tfie NC of the domain Changes in resources adjacent 

changing topology information. Such communication to the NC are recorded locally by the NQ and changes 

paths are established whenever two domains are adja- « resources adjacent to NNC I nodes ; in the = domain .of 

cent; that is, whenever there is a single communications ** NC are reported tothe NC by the NNC using the 

link connecting a node of one domain to a node in an- ownership path. The NNC reports the status of all rts 

other. In FIG 4, a virtual network of the domains iden- 40 connections on any change to a single connection. In 

tTied i FIG 1 is detected addition, it forwards the cause of its report (new re- 

1 t v u * J * i * ^ i . f , , source, failed resource, or change in characteristics of a 

In a hybrid network with automatic update of topol- ™^„ZL\ „„a i™i L„ „*™« * ™,™~o ;^„ t ; 

resource) and its local time stamp as a sequence ldenti- 

^Sy* fier 

(1) Each node monitors its local topology, 

(2) An owning NC is established for each NNC, 45 Establishing Communication Across Domains 

(3) Each NC learns and maintains the topology of its Part of ^ domain mfonna ti on retained by 
^^^kS? .^connections), a NC is the identification of nodes adjacent to its do- 

(4) Each NC in the virtual network of NCs exchanges main which are OWIied by another NC, and the identifi- 
domain topology mformation with other NCs in order of the owning NC xhrough this information, the 
to establish a complete network topology data base in NC of a domain establishes communication with each 
each NC. j*jC of adjacent domains at the earliest possible point. 

(5) All the above conditions are established/reestab- The path used by this communication consists of three 
lished upon changes (new additions of connections or parts: 

nodes, changes in status of existing connections or 55 j The path to a node in the NCs domain which is 

nodes). adjacent to the other domain. This path could be 

Some of the advantages of a hybrid network with auto- the NC itself if the NC is adjacent to the other 

matic update of topology are: domain. 

The network is insensitive to the order in which 2. The link connection between the domains, 

nodes are introduced. Communication is possible go 3. The path from the adjacent node in the other do- 

among any subset of nodes which are connected main to its owning NC along that ownership path, 

and include at least one NC. The routing tables of the nodes along this three part 

New nodes and links may be introduced without path are updated to allow subsequent communication 

disrupting the existing network. between the NCs of each domain. This communication 

The network can tolerate failures of nodes and links, 65 path between the two NCs is considered a link in the 

and still be able to establish communication among virtual network of NCs. 

any subset of nodes which remain connected and In order for each NC to obtain knowledge of the 

include at least one NC. complete network topology, it is necessary for it to 
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exchange domain topologies with all other NCs. Hence, 
an NC having a change in the topology of its domain 
broadcasts the topology of its entire domain to each 
adjacent NC in the virtual network of NCs. These adja- 
cent NCs broadcast the received topology to each of 
their adjacent NCs in the virtual network of NCs, and 
so forth until each NC has received the notification. 

Error Recovery Situations 

The loss of a connection between two nodes is re- 
ported to the domain's NC. If a portion of the domain is 
no longer connected with the portion controlled by the 
NC of the domain, the lost portion of the domain is 
removed from the domain topology at the NC receiving 
this information. The resulting domain topology is 
broadcast to all adjacent NCs in the virtual network of 
NCs. 

Upon recovery of a connection, the normal bring up 
procedures are followed. This includes the establish- 
ment of an owner for each unowned NNC in the recov- 
ered portion and the establishment of communication 
paths to the NCs in any newly adjacent domains. If the 
disconnected portion still had connections to other 
domains, other NCs may have assumed ownership of 
the nodes before the reestablishment of a connection 
from the NC of the prior domain. 

Detailed Description of Hybrid Network Operation 
In the detailed description below, certain notations 30 
are used. Lower case letters (e.g., i, j, k) signify NNCs, 
while upper case letters (e.g., J, K, M) signify NCs. In 
cases where the node could be either, lower case letters 
are used. 

The term OWNER® refers to the current owning 35 
NC of node L The owner of an NC is the NC itself. If 
node i has no owner, the OWNER(i) is "NONE". 

Node Data Bases 

Each NC or NNC maintains a routing table 107 or 40 
207 consisting of information to support transmittal of 
packets to destinations in the network. Table 1 is a 
sample of an implementation of a routing table which 
assumes each packet carries both the identification of 
the origin of the packet and the ultimate destination. 45 
Through the routing table, the next node to which the 
packet should be forwarded is identified. 

TABLE 1 



20 



25 



Sample Routing Tabic for NNC a 
Node NNC a 


Origin Node 


Destination Node 


Next Node 


A 


b 


c 


b 


A 


A 


A 


d 


B 


d 


A 


A 



50 



55 



10 

TABLE 2 



Sample Topology Data Base for NC A 
NodeNCA 



Node (Owner) 



Adjacent 
Node (Owner) 



Link Sequence 
Weight Number 



A(A) 
A(A) 
a(A) 
a(A) 
e(A) 
B(B) 
B(B) 
B(B) 



a(A) 
B(B) 
A(A) 
B(B) 
c(B) 
MA) 
a(A) 
b(B) 



A link status table 208 maintained by each NNC The 
complete link status table for NNC a is shown in Table 
3. Only information about the local links is retained in 
the link status table, which is a subset of the topology 
data base. 

TABLE 3 



Sample Link Status Table for NNC a 
Node NNC a 


Node (Owner) 


Adjacent 
Node (Owner) 


Link 
Weight 


a(A) 


A(A) 


9 


a(A) 


B(B) 


7 


a(A) 


c(B) 


4 



Table 2 depicts a topology data base of a typical NC 60 
(NNCs do not have topology data bases). For each 
node in the network, the topology data base has that 
node's identification, the owner of that node, the identi- 
fication of each link for that node, the weights associ- 
ated with the link, the adjacent node's identification, 65 
and the owner of that adjacent node. In addition a se- 
quence number is kept for each node to aid in identify- 
ing old or duplicate topology information. 



Message Formats 

Before detailing the automatic updating of a hybrid 
network, the formats of the messages exchanged be- 
tween nodes are described. 

General: All messages are preceded by a header re- 
cord used for packet switching. Each header record has 
both the origin node and destination node identified. 
Also, there is a unique identifier for each message type. 
Appended to the header is the message content 

OWNER© Message: This message identifies node i 
and the current owner of node i, OWNER®. It carries 
a time stamp for sequence identification. 

Link Status Table Message: This message is sent from 
NNC i to its owning NC J. The message describes the 
event which caused this report, and carries a copy of 
the entire current link status table of NNC i, plus a time 
stamp for sequence identification. 

Topology Data Base Message: This message is sent 
from NC I to NC J. The message carries the entire 
topology data base of NC I, sorted by domain. Included 
in the domain information is the individual time stamps 
for sequence identification of each domain's informa- 
tion. 

RQST OWNER Message: This message is sent from 
NC I to NNC j requesting (or inviting) that NC I be- 
come the owner of NNC j or that NNC j join the do- 
main of NC I. The proposed ownership path is identi- 
fied in the message. The OWNERQ) message is sent in 
reply if NNC j accepts. 

The following events occur for a typical node such as 
NNC i: 

1. NNC i is initialized (FIG. 5). 

2. Link to node j is activated with weight W (FIG. 6"). 

3. Link to node j is deactivated (FIG. 7). 

4. Link to node j changes weight to W (FIG. 8). 
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5. NNC i receives RQST OWNER(i)=K message 2. Link to node j is activated with weight W (FIG. 
(FIG. 9). 13). 

6. NNC i lost its ownership path (FIG. 10). 3. Link to node j is deactivated (FIG. 14). 

7 . NNC i receives OWNER(j)=K message (FIG. 4. Link to node j changes weight to W (FIG. 15). 

11). 5 5. NC I receives OWNER© Message (FIG. 16). 

Referring now to FIG. 5, the functions performed on 6. NC I receives Link Status Table Message from 

initialization of a NNC are iUustrated as including NNC j (FIG. 17). 

F1-F3. The first function (at Fl) clears the link status 7. NC I receives Data Base Topology Message from 

table 208 (see FIG. 3). Function F2 clears the routing N C J (FIG. 18). 

table 207 (see FIG. 3) and function F3 sets the identifi- 10 12 shows the events occurring at a control node 

cation of the owner to none or null. on lts initialization. Function F30 clears the topol- 

FIG. 6 shows the events that occur at a typical NNC ogy data base 105, function F31 clears the routing 

when a link from that node to an adjacent node j is toble W and function F32 identifies the ownership 

activated with weight W. Function F4 adds an identifi- of ^ node > as itself * 

cation of the new link and its weight to the link status 15 FIG - 13 shows the events which occur at a control 

table 208. Function F5 transmits a message indicating node when a link to an adjacent node j is activated with 

the owner of the node i to the neighboring node j. a Particular weight. Function F33 adds the new link and 

FIG. 7 shows the events which occur at a typical ,ts wei Sht to the topology data base 105. Function F34 

NNC when a link to a neighboring node is deactivated. sei \ ds a m ^sage indicating the ownership of the control 

Function F6 deletes the identification of that link from 20 ™ node * 

the link status table 208. Function F7 determines if the FIG. 14 shows the events which occur at a control 

node at which the event is occurring (node i) is owned. ak to an adjacent node j is deactivated. 

If it is not, no further action is necessary. On the other ^ c ?° n ™ mk * e to £? 10 ^ 

hand, if there is an owner, then F8 sends a link status „ bas **°f ^ nctl ° n F36 ^'^f / ^^node j is con- 

table message (identifying the remaining information in 25 nected to the network through other links, thjs^an be 

the link staL table 208) to the owner inf * ^2"? 1° 'T!™** b *f 

FIG. 8 shows the events that occur at a node i when ™ 110 other^ coimectmg node j to the 

vi* . ,« , , ♦ ™> t? network, then function F37 is performed to remove 

a linkto a neighboring ^node changes weight W. Func- nQdej frQm ^ , data ^ ^ function F38 is 

uonF9 ^changes the entry m ithe Imkstotus table 208 to 3Q performed to sendee topology database, as now modi- 
reflect the new weight for this link. Functions F10 and to adjacent domam s 

t0 /' unctions F8 ° f FI °- ? - • , On the other hand, if node j were still connected, 

FIG. 9 shows the events which occur at a typical function F39 determines if ^ node at wh ich 

NNC i on receipt of a message from a control node ihcsc m ^ is the owncr of node j. If 

requesting ownership. Function F12 determines if the 35 noU function « effected to inform other domains of 

node is already owned. If it is, function F13 is per- deactivation of the particular link, 

formed to essentially ignore the message. On the other <>„ ^ other tf ^ node j & ^ ownat of node 

hand, if the node i is not owned, then functions F14-F16 ^ then function F40 determines if the ownership session 

are performed. Function F14 writes a new entry defin- has been lost j/ the ownership session is still up (via a 

ing the ownership for the node i. Function F15 trans- 40 3ink other than the one which had been deactivated) 

nuts the contents of the node's link status table 208 to ^ function F 38 is performed. If the ownership session 

the now owning node K. Function F16 identifies the has been lost (because the ownership session path to the 

new owner of the node i to each adjacent node. n ode j included the deactivated link), function F41 

FIG. 10 shows the events which occur at a typical sends a request to become the owner from the node I to 

NNC i in the event that the communication path be- 45 the node j via a different link. Following function F41, 

tween it and its owner is disabled. This event could be function F38 is performed. 

triggered by the disablement of the control node, or any FIG. 15 shows the events which occur at a control 

other node or link between the NNC i and the owning no d e I when a link to node J changes weight Function 

node. Function F17 rewrites the identification of the F42 changes the link weight in the topology date base 

node's owner as none. Function F18 sends the owner- 50 105. Function F43, as a consequence of the change of 

ship message indicating that node i is no longer is the data in the topology data base 105 communicates the 

owned, to each adjacent node. contents of the topology data base 105 to adjacent do- 

FIG. 11 shows the events which occur at a node mains, 
receiving a message from an adjacent node as to the FIG. 16 shows the events which occur at a control 
adjacent node's ownership. Function F19 updates the 55 node I when it receives a message indicating the owner- 
link status table 208 with this new information. Funo ship of a node j. Function F44 compares the time stamp 
tion F20 determines if the node i is owned, and if not, no or sequence number of the message with the sequence 
other operation is necessary. On the other hand, func- number in the topology data base 105. If the message's 
tions F21 and F22 are performed to transmit the con- . sequence number indicates it is earlier than or the same 
tents of the link status table 208 of node i to its owner, 60 as data already in the topology data base 105, the infor- 
as a result of the change in the contents of the link status mation can be disregarded and no other events occur, 
table (F19). Function F22 transmits an ownership mes- e.g. the message is ignored. 

sage identifying the owner of node j, to the owner of the On the other hand, if the message time stamp shows 

node i. the message is later than data in the topology data base, 

VT - _ A . . . . 65 function F45 will update the topology data base 105 

NC Events and Actions witfl ^ contained in the message. Func 

The following events occur for the typical NC I: tion F46 checks to see if the owner of the node j is the 

1. NC I is initialized (FIG. 12). control node at which these events are occurring. If 
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that is the case, function F49 is performed to communi- It should be apparent from the foregoing that these 

cate the change in the topology data base of the control processes meet the requirements initially specified, e.g.: 

node I, to adjacent domains. On the other hand, if the 1. After some finite time all connected ordinary nodes 

owner of node j is not the control node I, then function will exist in one and only one domain and have an 

F47 checks to see if the node j is owned at all. If the 5 active session with their owning control node, so 

node j is not owned, then function F50 is performed to that routing information necessary at an ordinary 

attempt to obtain ownership of the node j. Following node can be retrieved from an identified control 

transmission of the request ownership message, function node through an identified and active path, 

F49 is performed. 2. Any change in the status of the network (activation 

On the other hand, if the owner of node j is a different 10 or deactivation of either a link or a node) which is 

control node, then function F48 is performed to deter- sensed by an ordinary node will be firstly commu- 

mine if the control node which is the owner of node j is nicated to the owning control node, and if that 

adjacent in the virtual network to the control node I at information is new to the control node, it will be 

which these events are occurring. If it is, then function communicated to adjacent domains. In this fashion, 

F49 is performed. On theother hand, if the owner of the 15 current topology information will tend to propa- 

node j is not adjacent to the node I in the virtual net- te the network, and 

work, then function F51 establishes a path to the own- 3. the event that failure of a node or a link tenni- 

ing node in the virtual network and function F49 is nates ^ ownership session, a new ownership ses- 

performed. sjon ^ ^ initiated with either the same or a dif- 

FIG. 17 shows the events occurring at a control node 20 ^ node b ^ cvcnt th at the origi- 

I on receipt of a link status table message from NNC j. ^ a ^ m the th induded in 

Function F52 compares Re sequence number or time ownership session has been deacti- 

stamp with link status table information previously re- ™ j p . .„ . . . , 

. *\7 j • *u * w,. A~* n w<» mc if tt.. vated, or a new ownership session will be mitiated 

ceived from node 1 m the topology data base 105. it the , ^ - i_ t_ j i_ j„ *: 

j &j ordinary node which had been deacti- 

message s sequence number is not newer, then the mes- 25 . . * 

sage is ignored. On the other hand, function F53 will _ vated - * re f tivated ' . . . . 4 . . 

modify lie topology data base 105 with a new line data The purpose for maintaining a topology data base in 

of status table from the node j, As a consequence of the the control node and continuous identification of an 

change in the contents of the topology data base 105, ownership session between each ordinary node and its 

function F54 is performed to transmit this information 30 owner is for the purpose of routing messages. The ad- 

to adjacent domains. vantage of the invention is that the extensive topology 

Finally, FIG. 18 illustrates the events occurring at a data base need not be maintained at each ordinary node, 

control node I receiving a topology data base message Thus, when a message is received at NNC i (a typical 

from a different control node J. Function F55 isolates ordinary node), for transmission to NNC j, node i may 

that portion of the message relating to the first domain 35 not have routing information for that particular mes- 

in the message. Function F56 compares the sequence sage. FIG. 19 illustrates the procedure for message 

number associated with this information with the se- routing at a typical node i. The procedure is entered 

quence number of the corresponding information exist- when a message is received identifying the destination 

ing in the topology data base 105. If the sequence num- node j. Function F70 accesses node i's routing table to 

ber existing in the data base is more recent than the 40 identify the path to node j. Function F71 determines if 

information in the message, then this portion of the a path is available. If it is, function F72 isolates the next 

message can be ignored and function F57 is skipped. On no de in the path and function F73 transmits the message 

the other hand, if the information in the isolated portion to the thus identified node. 

in the message is more recent than the information in j n e vent that the routing table of node i does not 
the topology data base 105, then function F57 is per- 45 contain a path to node j, then function F79 is performed 
formed to update the topology data base 105 with re- to identify owner (i). If node i is not owned, then func- 
spect to the domain's information. Function F58 deter- tion ^ performed to discard the message, since it 
mines if there is information in the message concerning cannot be routed. On the other hand if owner (i) exists, 
a different domain. Assuming there is, function F59 ^ ^4 ^ performed to request the path, 
isolates that portion of the message respecting the next 50 Function p7 5 determines if the response from owner (i) 
domain and functions F56 and F57 are again performed. is availab]e . when it is function F76 determines if that 
In effect functions F56, F57, F58 and F59 form a loop to ^ identifies ^ path . K ^ response does not 
repeatedly isolate information respecting a domain m ^ ^ ^ faction F77 is performed to 
the message and compare that f^J^^F^^ „ discard the message, since it cannot be routed. Alter- 
ing information existing m *e topology data base 105. 55 ^ ^ response from 
More current ir^ormauon is recorded, older mfonna- JJ£ then P fu nction F78 stores the path in the rout- 
tion is discarded. When the entire message has been : " , , ' r . ^ r nnA ™* 
proc^mthisfasWonthenfunctionFfiOdeter^ 

any changes have been made to the topology data base ™ performed to identify the next node m tte path and 
105. If no changes have been made no other function is 60 transmit the message to or at least toward that particu- 

necessary. On the other hand, if one or more changes lar node ' . , . 

have been made in the topology data base, then function Having thus described our invention, what we claim 
F61 is performed to transmit the contents of the topol- as new, and desire to secure by Letters Patent is: 

ogy data base to all adjacent domains except the domain 1. A method of maintaining a topology data base 
J from which the message was received. This transmis- 65 which is available for message routing in a dynamic, 
sion could include either the entire topology data base hybrid mesh connected network including at least one 
or only that portion of the topology data base which control node and a plurality of ordinary nodes, said 

was altered by function F57. method comprising the steps of: 
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(a) maintaining in each of said ordinary nodes a link 
status data base identifying directly connected 
nodes, and 

(b) maintaining in each of said control nodes a topol- 
ogy data base, by: 5 

(i) establishing one or more domains of nodes, each 
such domain including only one control node, 

(ii) communicating network status information from 
an ordinary node to the single control node of its 
domain in response to a change in network status 10 
sensed by said ordinary node, and 

(iii) altering the topology data base at said control 
node in response to information transmitted by said 
ordinary node, 

whereby each said control node is informed of net- 15 
work status changes adjacent to any node in its 
domain. 

2. The method of claim 1 wherein said step (i) com- 
prises: 

(ai) transmitting a message from an ordinary node, 
not included in any domain, to a neighboring node 
indicating the exclusion of said transmitting node 
from any domain, 

(bi) at said neighboring node, transmitting the infor- 
mation received in step (ai) to the single control 
node in the domain of the neighboring node or if 
the neighboring node is not within a domain stor- 
ing the received information until such time as the 
neighboring node joins a domain, ^ 

(ci) at a control node receiving the message of step 
(bi), transmitting an invitation to said ordinary 
node inviting said ordinary node to join the domain 
of the transmitting control node, 

(di) on receipt of the message of step (ci) at the ordi- 35 
nary node, altering its status to be within the trans- 
mitting control nodes domain by storing the iden- 
tity of the transmitting control node as the owner 
of the ordinary node and transmitting a message to 
the transmitting control node acknowledging ac- $q 
ceptance of the invitation. 

3. The method of claim 1 in which said step (i) com- 
prises: 

(ai) transmitting a message from an ordinary node, 
not included in any domain, to each neighboring 45 
node indicating the exclusion of said transmitting 
node from any domain, 

(bi) at each said neighboring node, transmitting the 
information received in step (ai) to the single con- 
trol node in the domain of each neighboring node, 50 

(ci) at any control node receiving the message of step 
(bi), transmitting an invitation to said ordinary 
node inviting said ordinary node to join the domain 
of the transmitting control node, 

(di) on receipt of the message of step (ci) at the ordi- 55 
nary node, altering its status to be within the do- 
main of the particular control node whose message 
is first received by storing the identity of the partic- 
ular control node as the owner of the ordinary 
node, transmitting a message to the particular con- 60 
trol node acknowledging acceptance of the invita- 
tion and on receipt of each other step (ci) message 
transmitted by other control nodes, ignoring each 
said message. 

4. The method of claim 2 or 3 which includes the 65 
further step of transmitting to each neighboring node of 
the ordinary node an indication that the ordinary node 

is now within an identified domain. 
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5. The method of claim 1 in which said step (b) in- 
cludes: 

(bi) identifying at each control node those other con- 
trol nodes adjacent the domain of the control node, 
(bii) repeating to said other control nodes at least the 
information communicated to said control node by 
said ordinary nodes in the domain of said control 
node respecting network status changes, 
whereby all said other control nodes are informed of 
said network status changes to thereby maintain an 
accurate topology data base in all said other control 
nodes. 

6. The method of claim 1 in which said link status 
data base in each of said ordinary nodes includes infor- 
mation identifying the control node of the domain of 
each adjacent ordinary node. 

7. The method of claim 1 in which said step (i) in- 
cludes the steps of: 

(ai) at each control node identifying an ownership 
session as a particular set of ordinary nodes 
through which messages to a particular ordinary 
node within the domain of the control node are 
transmitted, 

(ah) in response to information received at any con- 
trol node that a particular ownership session to said 
particular ordinary node is now unavailable delet- 
ing said unavailable ownership session from said 
topology data base, 

(aiii) deriving from said topology data base another 
set of nodes through which messages may be sent 
to said particular ordinary node, 

(aiv) transmitting an invitation to said particular ordi- 
nary node over said another set of nodes, and 

(av) if another set of nodes to said particular ordinary 
node is not contained within said topology data 
base, then instead of steps (aiii) and (aiv) deleting 
said particular ordinary node from said topology 
data base. 

8. The method of claim 1 in which said link status 
data base and said topology data base each include for 
each pair of adjacent nodes, an efficiency factor for 
transmissions between the adjacent nodes. 

9. The method of claim 8 in which communicated 
network status information includes changes in node 
adjacency or efficiency and is accompanied by a time 
stamp related to the time at which a particular change 
has occurred. 

10. The method of claim 9 in which said step (b) 
includes: 

(bi) identifying at each control node those other con- 
trol nodes adjacent the domain of the control node, 
(bii) repeating to said other control nodes at least the 
information communicated to said control node by 
said ordinary nodes in the domain of said control 
node respecting network status changes, 
whereby all said other control nodes are informed of 
said network status changes to thereby maintain an 
accurate network wide topology data base in all said 
other control nodes. 

11. The method of claim 10 in which said step (bi) 
includes transmitting the time stamp received from the 
transmitting ordinary node and which includes the fur- 
ther step of: 

(bin) updating the topology data base at one of said 
other control nodes by comparing the received 
time stamp with the corresponding time stamp in 
its topology data base to determine which of the 
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received or corresponding network status informa- 
tion is more current 

12. The method of claim 10 in which said step (bi) 
includes transmitting the time stamp received from the 
transmitting ordinary node and which includes the fur- 5 
ther step of: 

(biii) updating the topology data base at each of said 
other control nodes by comparing the received 
time stamp with the corresponding time stamp in 
its topology data base to determine which of the 10 
received or corresponding network status informa- 
tion is more current 

13. A method of routing a message at an ordinary 
node in a dynamic, hybrid mesh connected network 
which comprises the method of claim 1 and the addi- 15 
tional steps of: 

(c) acquiring routing information from the control 
node of the domain including said ordinary node, 
and 

(d) routing said message from said routing informa- 20 
tion. 

14. A method of maintaining a topology data base in 
a dynamic, hybrid mesh connected network including 
at least one control node and a plurality of ordinary 
nodes, said method comprising the steps of: 25 

(a) maintaining in each of said ordinary nodes a link 
status data base identifying only directly connected 
nodes, and 

(b) maintaining in each of said control nodes a topol- 
ogy data base, by: 30 



18 

(i) identifying at each control node those other con- 
trol nodes adjacent the control node, 

(ii) repeating to said other control nodes at least net- 
work status change information derived at said 
control node, 

whereby all said other control nodes are informed of 
said network status changes to thereby maintain an 
accurate topology data base in all said other con- 
trol nodes. 

15. The method of claim 14 in which said step (ii) 
includes: 

(iia) establishing one or more domains of nodes, each 
such domain including only one control node, 

(iib) communicating network status information from 
an ordinary node to the single control node of its 
domain in response to a change in network status 
sensed by said ordinary node, and 

(lie) altering the topology data base at said control 
node in response to information transmitted by said 
ordinary node. 

16. A method of routing a message at an ordinary 
node in a dynamic, hybrid mesh connected network 
which comprises the method of claim 15 and the addi- 
tional steps of: 

(c) acquiring routing information from the control 
node of the dore*"" including said ordinary node, 
and 

(d) routing said message from said routing informa- 
tion. 
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